Informative data streaming server

ABSTRACT

A method of data streaming from an encoder to a decoder is provided. The method includes establishing a connection from said encoder to said decoder; acquiring measurements characterizing said connection; estimating transmittance variation of said connection according to said measurements; determining a permissible encoding rate of said encoder compatible with the transmittance variation; determining a set of metrics characterizing said connection; receiving, at said encoder, a signal from a source; instructing said encoder to encode said signal according to said permissible encoding rate; selecting a set of notification-display intervals each corresponding to a respective metric among said set of metrics; and superimposing on said signal, during said notification-display intervals, corresponding indications of said metrics. A corresponding system is also provided.

The present application is a Continuation of U.S. patent applicationSer. No. 13/157,157 filed on Jun. 9, 2011, now issued as a U.S. Pat. No.9,231,992 on Jan. 5, 2016, which in turn is a continuation of U.S.application Ser. No. 12/463,410 filed on May 10, 2009, which has beenissued as a U.S. Pat. No. 7,975,063 on Jul. 5, 2011, the entire contentsof the above-noted applications and patents being incorporated herein byreference.

FIELD OF THE INVENTION Background

A streaming server may establish multiple connections through a networkfor transmitting multimedia signals to a number of clients. A connectionmay traverse multiple links and the transmission properties of theconnection may vary with time. To ensure preserving signal integrity asconnection-state varies with time, the streaming server may adaptsignals sent to a client to be compatible with current transmissionlimitations. In U.S. patent application Ser. No. 12/180,697, filed onJul. 28, 2008, methods for real-time connection-state evaluation andsignal-content adaptation were disclosed.

In addition to the effect of transmission path conditions, the actualflow rate of data transmitted to a client may also be influenced byconditions related to the streaming server or client's equipment. Somedata compression techniques may yield a data stream of a fluctuatingflow rate which follows temporal signal variations. A client may beaware of the actual flow rate of received data, which may differ from aflow rate specified in a connection request. However, the client has nomeans of determining the flow rate of data transmitted from thestreaming server or the cause of any discrepancy between the specifiedflow rate and the flow rate of received data. The flow rate of receiveddata may be less than the flow rate of data transmitted from thestreaming server due to data loss en route.

There is, therefore, a need to devise an informative streaming serverwhich informs each client of time-varying flow rate of transmitted dataand causes for changing the flow rate.

SUMMARY

The present invention provides a data-streaming server transmittingmultimedia data to a number of clients. The server determines apermissible flow rate to a client according to transmission-mediumconditions as well as conditions at both the client's equipment and thedata-streaming server. The server informs the client of causes of flowrate variations.

In accordance with one aspect, the present invention provides a methodof informative data streaming. The method is implemented in a streamingserver and comprises steps of: encoding, using an encoder coupled to thestreaming server, a multimedia signal having a plurality of signals ofdifferent signal types according to a signal-specific encoding rate toproduce multimedia data comprising a plurality of data components havinga one-to-one correspondence to the plurality of signals; transmittingthe multimedia data from the streaming server to a client over atelecommunication channel; and receiving feedback information from theclient. The signal-specific encoding rate is initially set to apredefined value corresponding to a respective signal.

Quality metrics of the telecommunication channel are determinedaccording to the feedback information, and a permissible bit rate foreach signal is determined according to the quality metrics. Thesignal-specific encoding rate is then adjusted to a value not exceedingthe permissible bit rate. A notification message indicating the qualitymetrics and the permissible bit rate is generated for each signal andsuperimposed on a selected signal from among the plurality of signalsconstituting the multimedia signal. Preferably, the notification messageis superimposed on the multimedia signal during a time window followinga relative change in the permissible bit rate exceeding a predeterminedvalue. The duration of the time window is an operational parameter.

The method further comprises determining, for each signal, an indicationof an acceptable bit rate of a decoder associated with the client anddetermining the respective initial value of the signal-specific encodingrate according to the acceptable encoding rate.

The method further comprises determining a mean value of thesignal-specific encoding rate over successive observation intervals and,when the mean value is smaller than a respective predeterminedthreshold, an indication of the mean value is superimposed on theselected signal.

The multimedia data may comprise real-time-transport-protocol (RTP)packets and the feedback information may be acquired fromreal-time-transport-control-protocol (RTCP) packets received from saidclient.

In accordance with another aspect, the present invention provides amethod of data streaming from an encoder of a streaming server to adecoder associated with a client of the streaming server. The methodcomprises steps of: establishing a connection from the encoder to thedecoder; acquiring measurements characterizing the connection;estimating transmittance variation of the connection according to themeasurements; and determining a favorable encoding rate of the encodercompatible with the transmittance variation.

Upon receiving a signal from a source, a flow controller of thestreaming server instructs the encoder to encode the signal according tothe favorable encoding rate. The flow controller determines a firstdifference between the favorable encoding rate and a previous encodingrate of the encoder. The value of the previous encoding rate may beinitialized to a nominal encoding rate. When an absolute value of thefirst difference exceeds a predetermined first threshold, a firstnotification-display interval is determined and indications of thecurrent encoding rate and the first difference are superimposed on thesignal during the first notification-display interval and the value ofthe previous encoding rate is reset to equal the favorable encodingrate.

The method further comprises observing the signal over distinctlydefined successive time intervals, determining a current frame rate ofthe encoder, and determining a second difference between the currentframe rate and a previous frame rate. The previous frame rate isinitialized to a nominal frame rate. When an absolute value of thesecond difference exceeds a predetermined second threshold, indicationsof the current frame rate and the second difference are superimposed onthe signal, during a second notification-display interval and theprevious frame rate is reset to equal the current frame rate.

The method further comprises extracting, from the measurements, anindication of current data-loss fraction and determining a thirddifference between the current data-loss fraction and an indication of aprevious data-loss fraction. The indication of a previous data-lossfraction is initialized to zero. When an absolute value of the thirddifference exceeds a predetermined third threshold, indications of thecurrent data-loss fraction and the third difference are superimposed onthe signal, during a third notification-display interval and theprevious indication of the data-loss fraction is reset to equal thecurrent indication of data-loss fraction.

The method further comprises extracting, from the measurements, anindication of a current transfer delay between the encoder and thedecoder and determining a fourth difference between the current transferdelay and a stored value of a previous transfer delay. The stored valueof the previous transfer delay is initialized to zero. When an absolutevalue of the fourth difference exceeds a predetermined fourth threshold,indications of the current transfer delay and the fourth difference aresuperimposed on the signal, during a fourth notification-displayinterval and the previous indication of the transfer delay is reset toequal the current indication of transfer.

The method further comprises extracting, from the measurements, anindication of occupancy of a receiving buffer at the decoder anddetermining a fifth difference between the current occupancy and astored value of a previous occupancy of the receiving buffer. The storedvalue of the previous occupancy is initialized to zero. When an absolutevalue of the fifth difference exceeds a predetermined fifth threshold,indications of the current occupancy and the fifth difference aresuperimposed on the signal, during a fifth notification-display intervaland the previous indication of the occupancy is reset to equal thecurrent indication of the occupancy.

The first, second, third, fourth, and fifth notification intervals donot overlap. The five notification intervals may be allocated indifferent time frames successively separated by an integer multiple oftime-frame periods so that they may be displayed individually one at atime. Alternatively, the five notification intervals may be allocatedwithin a common time frame so that they may be displayed ensemble.

In accordance with a further aspect, the present invention provides aninformative streaming server computer, to be also referred to asstreaming server. The server comprises: a signal-interface devicecommunicatively coupled to a plurality of signal sources providing aplurality of signals of different signal types; a processor; and acomputer readable medium storing executable instructions thereon. Theinstructions causes the processor to perform functions of: an overlaymodule for superimposing notification data on selected signals fromamong the plurality of signals; an adaptive encoder for receiving theplurality of signals from the overlay module and producing a pluralityof data streams, having a one-to-one correspondence to the plurality ofsignals, according to a specified encoding rate for each signal of theplurality of signals; a probing module for generating probing packets; anetwork interface for transmitting the data streams and the probingpackets to at least one client from among a plurality of clients andreceiving response packets from the at least one client; and a flowcontroller. The flow controller analyzes the probing packets and theresponse packets to determine a permissible encoding rate for eachsignal and instructs the adaptive encoder to encode each signalaccording to the respective permissible encoding rate. The flowcontroller presents an indication of the new encoding rate to theoverlay module for inclusion in the notification data superimposed onthe signal.

The informative streaming server further comprises a selector coupled tothe signal-interface device. The selector directs the selected signalsto the overlay module and directs any remaining signals from theplurality of signals, not subject to notification overlay, directly tothe adaptive encoder.

The encoder determines variations of a signal received from the overlaymodule over successive frame periods and produces encoded datarepresenting the signal. The encoder communicates the encoded data tothe client through the network interface. The encoder operates underconstraint of a permissible bit rate received from the flow controller.The frame period is predetermined and provided to the encoder.

The computer readable medium further stores instructions which, whenexecuted, cause the processor to determine mean values of bit rates ofthe encoded data over successive observation intervals and communicate aspecific mean value of bit rate over an observation interval to theoverlay module for inclusion in the notification data when the specificmean value is less than a predetermined fraction of the permissible bitrate. An observation interval comprises a predetermined integer multipleof a frame period.

According to an embodiment, the probing module stores instructionswhich, when executed, cause the processor to formulatereal-time-transport-protocol (RTP) packets and interpretreal-time-transport-control-protocol (RTCP) packets.

The adaptive encoder may comprise multiple encoding modules forproducing the plurality of data streams, and a combining module forinterleaving the data streams. Each encoding module encodes signals of arespective signal type. The signal types may include audio signals,video signals, still images, animations, and texts. The overlay modulemay comprise multiple overlay units, each for superimposing anotification compatible with a specific signal type.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be further described withreference to the accompanying exemplary drawings, in which:

FIG. 1 illustrates a streaming server communicatively coupled to anumber of clients, in accordance with an embodiment of the presentinvention;

FIG. 2 details the streaming server and a connection to one of theclients, in accordance with an embodiment of the present invention;

FIG. 3 illustrates components of the streaming server, in accordancewith an embodiment of the present invention;

FIG. 4 illustrates an overlay module in the streaming server of FIG. 3,the overlay module comprising multiple overlay units, in accordance withan embodiment of the present invention;

FIG. 5 illustrates an adaptive encoder in the streaming server of FIG.3, the adaptive encoder comprising multiple encoding modules, inaccordance with an embodiment of the present invention;

FIG. 6 illustrates an encoding module in the adaptive encoder of FIG. 5,the encoding module comprising a tracking module and a stream-formationmodule, in accordance with an embodiment of the present invention;

FIG. 7 illustrates a decoder associated with a client, in accordancewith an embodiment of the present invention;

FIG. 8 illustrates temporal variation of encoding rate of a signal foruse in an embodiment of the present invention;

FIG. 9 illustrates a process of determining a permissible encoding rateof a signal, according to connection characterization, and overlaying anindication of the permissible encoding rate on the signal, in accordancewith an embodiment of the present invention;

FIG. 10 illustrates a process of tracking temporal variation of anencoding rate of a signal and overlaying an indication of the temporalvariation on the signal, in accordance with an embodiment of the presentinvention;

FIG. 11 illustrates allocation of notification-display intervals, inaccordance with an embodiment of the present invention;

FIG. 12 illustrates the use of a standard protocol for communicationbetween the streaming server and a client;

FIG. 13 illustrates a process of determining favorable encoding rate ofa signal using a standard protocol and overlaying an indication of thefavorable encoding rate on the signal, in accordance with an embodimentof the present invention;

FIG. 14 details a step of extracting connection characterizing data fromfeedback signals received from a client, in accordance with anembodiment of the present invention; and

FIG. 15 details a step of determining a permissible encoding rate, inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION Terminology

Encoding rate: The bit rate of an encoded signal may fluctuate with timeand the bit rate of the encoded signal is herein called encoding rate.

Adaptive encoder: An adaptive encoder is a device capable of encoding asignal to a specified maximum flow rate within a predefined flow-raterange. The encoding attempts to maximize encode-signal fidelity. Theterms “encoder” and “adaptive encoder” are used synonymouslyhereinafter.

Nominal flow rate: A flow rate negotiated between a client and astreaming server, or determined automatically by a streaming serverbased on knowledge of client's equipment, is herein called a nominalflow rate.

Encoding coefficient: An encoding coefficient, denoted E, is a scalingfactor which is multiplied by the nominal flow rate to determine apreferred flow rate compatible with a current state of a connectionbetween a streaming server and a client.

Connection transmittance: The maximum flow rate which can be sustainedby a connection from a streaming server to a client without violatingpreset transmission performance objectives is herein called a connectiontransmittance. The actual flow rate may be less than a connection'stransmittance due to factors other than the physical path providing theconnection.

Performance characteristic: Performance characteristics are definedherein as measurable connection properties such as transfer-delayvariation, data-loss proportion, signal distortion, etc.

Statistic: A statistic is a value (such as a mean value) derived from aset of data samples.

Metric: A Metric is a measure of a quality of a path or a connection ina telecommunication network. A Metric may relate to a connectionproperty such as latency, reliability, capacity, etc., of a path orconnection within a path.

Downstream control packet: A control packet sent from a streaming serverto a client is a downstream control signal.

Upstream control packet: A control packet sent from a client to astreaming server is an upstream control signal.

Frame period: A multimedia signal typically includes a video signal. Avideo signal is organized into frames of equal frame periods.

Observation interval: Feedback signals received at a streaming serverfrom a client are preferably examined during observation intervals(observation periods) of appropriate durations to detect changes ofconnection state. The observation intervals (observation periods) may besuccessive or selected according to any other temporal order.

Operational parameters: The streaming server may store nominal values ofsystem parameters such as encoding rates for different signal types, thelength of an observation interval for determining a mean value ofencoding rate, duration of each notification of a specific notificationtype, a specified period for optional periodic insertions ofnotification data, or the length of a frame period of a video signal. Aconnection request from a client may also specify operational parametersfor the connection.

Gradient: Conventionally, the slope of a continuous function relatingtwo variables is known as a gradient. In the case of a sampled function,the gradient may be approximated by determining the slope from a smallnumber of samples. Herein, the gradient is the slope of measurementsamples determined over a period covering a small number of samples.

Regression line: A straight line drawn through a set of data anddetermined according to some criterion, such as minimizing the sum ofsquares of data deviation from the straight line, is a regression line.The slope of a regression line may reliably indicate a trend of the dataif the data dispersion meets certain conditions.

Real-time transport protocol (RTP): RTP defines a standardized packetformat for delivering audio and video over the Internet (defined by theInternet Engineering Task Force (IETF), RFC 3550).

Real-time transport control protocol (RTCP): RTCP provides out-of-bandcontrol information for an RTP flow and is used periodically to transmitcontrol packets in a streaming session. The primary function of RTCP isto provide feedback on connection quality.

FIG. 1 illustrates a streaming server 120, or a streaming servercomputer 120, communicatively coupled to a number of clients 160 througha network 180. The streaming sever 120 transmits multimedia signals tothe clients 160 and receives feedback information from the clients. Inaccordance with one embodiment, the real-time-transport protocol (RTP)is used for downstream communications from the multimedia server 120 tothe clients 160 and the associated real-time-control-transport protocol(RTCP) is used for exchanging control information between clients 160and the streaming server 120.

FIG. 2 details the streaming server 120 and downstream communicationwith a client 160. The server 120 comprises an encoder 240 receivingmultimedia signals from signal sources 220 either directly or through anoverlay module 230. The overlay module 230 may superimpose informationrelevant to connection condition on multimedia signals received from thesignal sources 220. A network interface 250 adapts encoded signals fortransmission over a link 262 to network 180 and transmission from thenetwork 180 to the client 160 over a link 264. The server 120 alsoincludes a flow controller 260 for receiving feedback information fromthe client 160 through link 284 from the client 160 to the network 180and link 282 from the network 180 to the flow controller 260 throughnetwork interface 250. The flow controller 260 is coupled to the encoder240 and the overlay module 230. The flow controller 260 analyzes thefeedback information to characterize the downstream connectiontraversing link 262, the network 180, link 264, and the client'sequipment. The feedback information is categorized to indicate differentperformance aspects of the downstream connection. The flow controller260 determines a permissible transmission rate over link 262 inaccordance with the received feedback information. The encoder 240encodes the multimedia signals received from the signal sources 220,either directly or through the overlay module 230, at encoding ratescollectively not exceeding the permissible transmission rate determinedaccording to the feedback information.

FIG. 3 illustrates further details of the informative streaming server120. The streaming server 120 comprises a master processor 350 and acomputer readable storage medium 352 having computer executableinstructions stored thereon, which are organized into instruction sets,each instruction set stored in the computer readable medium forming acorresponding functional unit, or module of the streaming server 120.

The functional units include overlay module 230, encoder 240, flowcontroller 260, a probing module 340, and network interface 250. Themaster processor 350 may comprise a number of processing units operatingconcurrently to share the processing workload.

The streaming server 120 has a source interface 316 for receivingmultimedia signals from a multimedia signal source 312. The multimediasignal source 312 may have multiple signal sources 320 providing signalsof different signal types, such as video, audio, animation, and textsignals. The signal sources are individually identified as 320(j),0≦j<m, m being equal to four in the exemplary multimedia signal source312.

Notably, a signal source 320 may comprise non-volatile memory devices,digital versatile discs (DVD), compact discs (CD), or floppy disks. Asignal source 320, as illustrated in FIG. 3, may be a local source ormay also be a representation of a data stream received from a remotestreaming server (not illustrated) which is re-encoded in encoder 240 ofthe streaming server of FIG. 3 and is subject to notification overlay inoverlay module 230 according to the methods described hereinafter.

The source interface 316 presents multimedia signals to encoder 240either directly or through overlay module 230. A selector 322 connectsthe source interface 316 to either the overlay module 230 through link324 or to the encoder 240 through link 330. A probing module 340implements steps of a protocol for monitoring connection conditions. Thenetwork interface 250 combines probing data from probing module 340 withencoded signals received from encoder 240 over link 342 for transmissionto a client 160 over link 262 to network 180. The network interface 250receives feedback data from clients 160 through network 180 and link 282and directs the feedback data to flow controller 260. The exchange ofdata among the components of the streaming server 120 uses methods wellknown in the art. Notably, the indicated paths between components of theinformative streaming server 120 are logical paths within a computingdevice (streaming server 120).

The computer readable storage medium 352 associated with the streamingserver 120 may comprise non-volatile memory devices, digital versatilediscs (DVD), compact discs (CD), or floppy disks, having computerreadable instructions stored thereon. The computer readable instructionsare executed by the master processor 350 to cause the overlay module230, the encoder 240, the flow controller 260, and the probing module340 to perform the functions described hereinafter.

The streaming server 120 receives multimedia signals from at least onemultimedia source 320 through a signal interface 316. The signalinterface 316 is a device for acquisition of signals from local sourcesor remote sources. The signal interface 316 includes respectivecomputer-executable instructions stored in a computer readable storagemedium, and may be a separate entity or embedded in the streamingserver. Only one multimedia source 312 is illustrated. The overlaymodule 230 superimposes notification data on the signals received frommultimedia source 312 during selected time intervals. The notificationdata are produced primarily by flow controller 260, based on informationreceived through network interface 250 and probing module 340 orencoder-state information received from the encoder 240 through link365, and submitted to the overlay module 230 through link 362. However,the encoder may generate specific notification data related to waitingencoding tasks or variation of encoding rates due to signal timevariation and submit the specific notification data directly to theoverlay module 230 through link 332. The encoder 240 receives signalsfrom the multimedia source 312 either directly through selector 322 andlink 330 or through the overlay module 230 and link 334. The encoder 240produces data streams having a one-to-one correspondence to the receivedsignals. The setting of the selector 322 to connect to the encoder 240or to the overlay module 230 may be included in a set of operationalparameters of the streaming server 120 and may be signal specific. Forexample, selector 322 may direct video signals of a specific encodingscheme to the overlay module 230 while directing other signals directlyto the encoder 240. Individual encoding rates for signals of differenttypes may be specified as operational parameters. The flow controller260 may adjust the individual permissible encoding rates according toconnection conditions received through network interface 250 and currentload conditions of the encoder 240 received through logical link 365.The flow controller instructs the encoder 240 to encode signals underthe constraints of the permissible encoding rates. Concurrently, theflow controller 260 provides indications of the permissible encodingrates and relevant causes to the overlay module 230 for signals directedfrom the source interface (signal interface) 316 to the overlay module230. The probing module 340 generates probing packets which may becombined with the output of encoder 240 in the network interface 250 orin a separate module.

The network interface 250 transmits data streams comprising payload dataand probing data to clients 160. The network interface also receivesdata-streaming requests as well as response packets providing feedbackinformation from clients 160. The flow controller 260 analyzes responsepackets and corresponding probing packets to evaluate connectionconditions and determine a permissible encoding rate for each signal.The flow controller instructs the encoder 240, through logical link 364,to encode each signal according to a corresponding permissible encodingrate and presents an indication of new encoding rates to the overlaymodule 230 through logical link 362, for inclusion in the notificationdata to be superimposed on multimedia signals before encoding.

A multimedia signal may comprise signals of different signal types. Theencoder 240 organizes video signals into frames of a specified frameperiod. The frame period may be entered as an operational parameter. Theflow controller 260 may determine an adjusted frame period according toconnection conditions or conditions at the encoder 240. The encoder 240determines variations of a signal received from the overlay module oversuccessive frame periods and produces encoded data, representing thesignal, under constraint of a permissible bit rate received from theflow controller.

The computer readable medium 352 may also store tracking instructionswhich, when executed, cause said processor 350 to determine mean valuesof bit rates of encoded data over successive observation intervals,where each observation interval covers a predetermined integer multipleof a frame period. When a specific mean value of bit rate over anobservation interval is found to be less than a predetermined fraction,for example ¼, of the permissible bit rate, an indication of thespecific mean value is communicated to the overlay module 230 forinclusion in said notification data. The tracking instructions may beassociated with the instruction set implementing the function of flowcontroller 260, or the instruction set implementing the function of theencoder 240.

The streaming server 120 implements a method of informative datastreaming where a client 160 receiving multimedia data from thestreaming server is informed of the encoding rate and causes of anychanges of the encoding rate.

Multimedia source 312, which comprises a number m≧1 of signal sources320 individually identified as 320(0) to 320(m−1), produces a multimediasignal having a plurality of signals of different signal types. Theencoder 240 encodes the multimedia signal according to a signal-specificencoding rate, initially set to a respective predefined value. Thepredefined encoding rates may be provided within a set of operationalparameters. The encoder 240 produces multimedia data comprising aplurality of data components having a one-to-one correspondence to the msignals 320(0) to 320(m−1).

The multimedia data is transmitted, through network interface 250, to aclient 160 over a telecommunication channel 262. The network interface250 receives feedback information from the client 160 and directs thefeedback information to the flow controller 260. The flow controller 260determines quality metrics of the telecommunication channel according tothe feedback information and determines, for each signal of the msignals produced by signal sources 320(0) to 320(m−1), a permissible bitrate according to respective quality metrics. The flow controller 260instructs the encoder 240, through logical link 364, to adjust each ofthe signal-specific encoding rates to a value not exceeding a respectivepermissible bit rate determined by flow controller 260. The flowcontroller 260 also produces notification signals each indicating thequality metrics and the permissible bit rate for each signal andpresents the notification signals to the overlay module 230 throughlogical path 362. The overlay module 230 superimposes the notificationsignals on the multimedia signal.

The feedback information received from a client 160 may include anindication of an acceptable signal-specific bit rate of a decoderassociated with the client equipment. The flow controller 260 uses theacceptable signal-specific bit rate to determine respective initialvalues of the signal-specific encoding rates.

The encoder 240 comprises computer-executable instructions stored in acomputer readable storage medium, which determine a mean value of thesignal-specific encoding rate over successive observation intervals, aswill be described with reference to FIG. 8. The length of an observationinterval may be set as an operational parameter. When the encoderdetermines that the mean value is smaller than a predetermined thresholdcorresponding to a given signal from among the m signals constitutingthe multimedia signal, the encoder informs the overlay module 230(through logical path 332) of an indication the mean value to besuperimposed on the multimedia signal before encoding.

The multimedia data may be organized according to any desired datastructure. It is convenient, however, to use standardized structuressuch as the real-time-transport-protocol (RTP) packets. Likewise, thefeedback information from clients 160 may be structured arbitrarily, butit is preferable to use a standardized structure such as that of thereal-time-transport-control-protocol (RTCP).

The superposition of informative signals over the multimedia signal maybe activated during a time window following a significant relativechange in the permissible bit rate where a relative change is consideredto be significant when it exceeds a predetermined value.

The overlay module 230 may comprise multiple overlay units each forsuperimposing a notification compatible with a specific signal type. Anoverlay module comprises a set of computer-executable instructionsstored in computer readable storage medium, which are applicable foroverlaying notification data for a signal of a specific type. FIG. 4illustrates an exemplary overlay module 230 comprising four overlayunits 430 individually identified as 430(0), 430(1), 430(2), and 430(3)for overlaying notification data on signals of different signal types.

As illustrated in FIG. 3, an overlay module 230 receives signals from amultimedia signal source 312 through source interface 316, selector 322,and link 324. Link 324 may carry a multiplex, temporal or spatial, ofseveral signals. FIG. 4 illustrates a link 324 having four logical paths424, individually identified as 424(0), 424(1), 424(2), and 424(3) tooverlay units 430(0), 430(1), 430(2), and 430(3), respectively.

An overlay module 230 receives notification data from flow controller260 through link 362 (FIG. 3). Link 362 may carry a multiplex, temporalor spatial, of several notification types. FIG. 4 illustrates a link 362from flow controller 260 having four logical paths 462, individuallyidentified as 462(0), 462(1), 462(2), and 462(3) to overlay units430(0), 430(1), 430(2), and 430(3), respectively.

An overlay module 230 may also receive specific notification data fromencoder 240 through link 332 (FIG. 3). Link 332 may carry a multiplex,temporal or spatial, of several signal-specific notification types. FIG.4 illustrates a link 332 from encoder 240 having four logical paths 432,individually identified as 432(0), 432(1), 432(2), and 432(3) to overlayunits 430(0), 430(1), 430(2), and 430(3), respectively.

An overlay module 230 sends signals combining payload signalsoriginating from multimedia signal source 312 and respectivenotification data, if any, to encoder 240 through link 334 (FIG. 3).Link 334 may carry a multiplex, temporal or spatial, of several signals.FIG. 4 illustrates a link 334 comprising four logical paths 434,individually identified as 434(0), 434(1), 434(2), and 434(3) fromoverlay units 430(0), 430(1), 430(2), and 430(3), respectively, toencoder 240.

An encoder 240 may comprise several encoding modules for encodingsignals of different types. FIG. 5 illustrates an exemplary encoder 240having four encoding modules 540 individually identified as 540(0),540(1), 540(2), and 540(3), each for encoding signals of a respectivesignal type. Encoder 240 receives signals through either link 330 orlink 334. In the exemplary encoder 240 of FIG. 5, either link 330 orlink 334 may comprise four logical paths each carrying a signaloriginating from multimedia signal source 312. Notably, any of the fourpaths, individually identified as 530(0), 530(1), 530(2), and 530(3) maybelong to either link 330 or link 334. For example, paths 530(0) and530(1) may be paths 434(0) and 434(1) of FIG. 4, carrying signals withsuperimposed notification data inserted in overlay module 230 whilepaths 530(2) and 530(3) may carry signals in their original formswithout inserted notifications.

The encoder 240 receives instructions from flow controller 260 throughlink 364. In the exemplary encoder 240 of FIG. 5, link 364 comprisesfour logical paths 564, individually identified as 564(0), 564(1),564(2), and 564(3) directed to encoding modules 540(0), 540(1), 540(2),and 540(3), respectively.

As described with reference to FIG. 4, encoder 240 may send specificnotification data to overlay module 230 through link 332 (FIG. 3) whichhas four logical paths 432(0), 432(1), 432(2), and 432(3) to overlayunits 430(0), 430(1), 430(2), and 430(3), respectively. The logicalpaths 432(0), 432(1), 432(2), and 432(3) originate from encoding modules540(0), 540(1), 540(2), and 540(3), respectively. An encoding module540(j), 0≦j<4, may process an original signal, received over link 330,or an annotated signal received over link 334 and designated to carrynotifications; only a logical path 432(j), 0≦j<4, from encoding modules540(j) handling an annotated signal is actually used. An annotatedsignal is a signal from one of signal sources 320(0) to 320(m−1) onwhich notification data are superimposed in overlay module 230.

Each encoding module 540(j), 0≦j<4, produces an individual data stream542(j), 0≦j<4, of a respective type (for example, a data streamrepresenting an audio signal, a video signal, a text, etc.). The fourdata streams 542(j), 0≦j<4, are interleaved in a combining module 580 toproduce a multimedia stream presented to network interface 250 throughlink 342. The combining module 580 interleaves encoded signals ofdifferent types according to any of interleaving methods known in theart. The signal types may include all, or a subset of, an audio signal,a video signal, a still image, an animation, and a text.

FIG. 6 illustrates one of encoding modules 540(j), 0≦j<4, comprising atracking module 620 and a stream-formation module 640. The index “j” inFIG. 6 identifies an encoding module 540(j) devised for signal encodingtechniques which capture signal variations over consecutive time framesand produces a data stream accordingly. Encoding modules employing otherencoding techniques may have different structures.

The tracking module 620 receives a signal over path 530(j), illustratedin FIG. 5, and determines signal variation over a designated period oftime, such as a frame time of a video signal. The tracking module 620produces an intermediate signal capturing signal variations. Thetracking module 620 may report indications of unusually low signalvariations, through a path 432(j), to overlay module 230 which maysuperimpose a respective notification as described above. Thestream-formation module 640 processes the intermediate signal receivedfrom tracking module 620, under an encoding-rate constraint receivedfrom flow controller 260 over logical path 564(j), to produce a datastream 542(j) to be offered to combining module 580.

FIG. 7 illustrates exemplary client equipment comprising aclient-network interface 720 for receiving a carrier signal from link264 and presenting the received carrier signal to a receiver-transmitter726 which demodulates the carrier signal to extract multimedia data sentfrom the streaming server 120. Packets of the demodulated multimediadata are placed in a buffer 738 at input of a decoder 740. The decoder740 and buffer 738 are operated under control of client controller 750.The decoder 740 processes the multimedia data to produce signals, suchas acoustic signals, video signals, animation-display signals, textcodes, etc., suitable for presentation to a user 780 through a userinterface 760. Feedback information is transmitted to the streamingserver 120 through client-network interface 720 and channel 284.

FIG. 8 illustrates temporal variation of encoding rate of an exemplarysignal constituting a component of a multimedia signal. In the exampleof FIG. 8, the permissible encoding rate 824 is determined by the flowcontroller 260 to be lower than the encoder's bit-rate limit 822. Thesignal is encoded at a rate which depends on the signal's temporalvariation, or equivalently the signal's spectral content, as viewed overa suitable observation time interval. For example, a video signalreceived from a source and organized in time frames may be encoded toretain only frame-to-frame changes and corresponding pointers to encodedsignal portions. Thus, the signal bit rate 826 required for properreproduction of the source's video signal may vary with time. Theencoding bit rate 828 may be slightly higher than the required signalbit rate 826 due to encoding overhead. When the encoding rate is low foran extended period of time due to small frame-to-frame variation, it isdesirable to inform a respective client of the reason. For example, ifthe encoding rate is less than ¼ of the permissible encoding rate for aperiod of more than a minute, and assuming that a user 780 (FIG. 7) hasmeans for determining received bit rate, a notification may be sentinforming the user 780 that the low encoding rate is caused by the slowvariation of the signal, rather than transmission conditions, encodercapability, or encoder overload.

Several disciplines of operating the data streaming system of FIG. 2 maybe devised within the operational capabilities of the streaming server120 and client equipment 160. The streaming server 120 receives aconnection request from a client 160 through network interface 250.After establishing a connection from encoder 240 of streaming server 120to decoder 740 of a client 160, the streaming server accesses amultimedia signal source 312 through source interface 316. The next stepis to acquire measurements characterizing the connection and estimatetransmittance variation of the connection based on the measurements. Asdefined above, the transmittance of a connection is the maximum flowrate which can be sustained by the connection from the streaming server120 to a client 160 without violating preset transmission performanceobjectives. A permissible encoding rate for each constituent signal ofthe multimedia signal, compatible with the transmittance variation, isthen determined.

FIG. 9 illustrates a process of determining a permissible or favorableencoding rate of a signal according to connection characterization andoverlaying an indication of the permissible encoding rate on the signal.The encoder 240 receives a signal from a signal source and initiallysets a current encoding bit rate for the signal to equal anencoder-defined limit (step 920). The encoder 240 sends the encodedsignal to the network interface 250. The probing module 340 generates aprobing signal and sends the probing signal to the network interface250. The network interface 250 combines the probing signal with theencoded signal and transmits the resulting signal (step 922) to thereceiver 726 of client 160 through links 262 and 264. The flowcontroller 260 receives feedback information from the client throughlink 282 (step 924). The feedback information includes, amongst otherinformation, measurements that characterize the transmittance of theestablished connection.

The flow controller 260 computes a permissible encoding rate for thesignal (step 926) and determines a difference between the permissibleencoding rate and the current encoding rate (step 928). If the magnitudeof the difference does not exceed a predefined value, as determined instep 930, the current encoding rate remains unchanged and step 922 isrevisited to send a new signal to the client. Otherwise, when themagnitude of the difference exceeds the predefined value, the flowcontroller 260 instructs the encoder 240 to modify the encoding rateaccording to the permissible encoding rate (step 932) and simultaneouslyinstructs the overlay module 230 to insert an indication of the changein encoding rate in the signal received from the signal source 220 sothat the encoder 240 encodes the combined signal (step 936). Step 922 isthen revisited. The current bit rate is set to equal the permissible bitrate for future reference (step 934). The flow controller may alsodetermine a notification-display interval (step 935) during which theindication of encoding-rate change is to be inserted.

The process of FIG. 9 results in displaying indications of encoding-ratechanges at a client's display device having a display screen or othermeans for presenting notification information. It may also be desirableto display other details associated with encoding-rate changes using theclient's display device. For example, an encoding-rate change of a videosignal may be realized by changing the frame rate. A display offrame-rate change may only be warranted when the change is significant.Thus, a second difference between a current frame rate, corresponding tothe permissible encoding rate, and a previous frame rate is determinedand display of frame-rate change is triggered when a magnitude of thesecond difference exceeds a certain value. The frame-rate display mayappear together with the encoding-rate display at the client's displaydevice. Alternatively, the frame-rate display may succeed theencoding-rate display after a sufficient interval so that at most one ofthe two displays is present at a given time. Thus, a secondnotification-display interval, not overlapping the firstnotification-display interval, is selected so that the twonotification-display intervals are either in a same frame or in separateframes.

It may also be desirable to display a cause of encoding-rate change.Thus, a current data-loss fraction may be determined from themeasurements, and a third difference between the current data-lossfraction and a previous data-loss fraction is determined. Where anabsolute value (magnitude) of the third difference exceeds apredetermined third threshold, a third notification-display interval isdetermined, during which indications of the current data-loss fractionand the third difference are superimposed on the multimedia signal atoverlay module 230. An indication of the previous data-loss fraction isupdated to equal the current data-loss fraction for use in decidingfuture notifications.

Likewise, a current transfer delay between the streaming server 120 anda client 160 may be determined from the measurements and a fourthdifference between the current transfer delay and a previous transferdelay may be determined. Where the magnitude of the fourth differenceexceeds a predetermined delay-variation threshold, a fourthnotification-display interval is selected for inserting indications ofthe current transfer delay and the fourth difference at overlay module230. The indication of the previous transfer delay is updated to equalcurrent transfer delay for use in deciding future notification displays.

A current occupancy of a receiving buffer at the decoder may bedetermined from the measurements extracted from the feedbackinformation. A fifth difference between the current occupancy and aprevious occupancy of the receiving buffer is then determined. Where themagnitude of the fifth difference exceeds a predeterminedoccupancy-variation threshold, a fifth notification-display interval isselected for superimposing on the multimedia signal indications of thecurrent occupancy and the fifth difference. An indication of theprevious occupancy is updated to equal the current occupancy for use indeciding future notification displays.

The above notification displays are only exemplary. Other notificationsmay be of interest and the overlay module 230 may be instructed tosuperimpose only a subset of available notification data.

If all the five notifications are selected, then the firstdisplay-notification interval, the second display-notification interval,the third display-notification interval, the fourth display-notificationinterval, and the fifth display-notification interval may be eitherallocated in different time frames, successively separated by an integermultiple of time-frame periods to appear at different times on a displaydevice, or allocated within a common time frame to appear together onthe display device.

The overlay module 230 may insert notifications of encoding rate changeand other related notifications in response to detecting a significantencoding-rate change. Alternatively, the overlay module 230 may insertthe notifications periodically, for example after every specifiedperiod, where a specified period may be defined in time units (20seconds for example) or defined in terms of a number of successiveframes, one thousand frames for example. The specified period may beincluded in operational parameters stored in the storage medium 352 orany other memory device.

FIG. 10 illustrates a process of tracking temporal variation of anencoding rate of a signal (FIG. 8) and overlaying an indication of thetemporal variation on the signal. The tracking process may beimplemented according to computer readable instructions stored in amemory device, which may be part of the storage medium 352. Theinstructions may be executed to cause processor 350 to implement thesteps of FIG. 10. The instructions may be included within the encoderinstruction set of encoder 240 or within the flow controller instructionset of flow controller 260.

A video signal is organized into frames and displayed at a predeterminedframe rate, such as 40 frames per second. Video encoding may beperformed according to any of several standardized methods. Typically,only variations between successive frames are transmitted todestination. Thus, the transmitted data varies according the videosignal temporal variation. It is useful to inform a client receiving anencoded video signal of the encoding rate at source. The mean encodingrate may be measured over predefined intervals, each interval covering aframe range having an integer number of frames. The number of data unitssent over an interval may be used to determine a mean encoding rate overthe interval. A data unit may be selected to be a bit, a byte, or aspecified number of bytes. To determine the actual encoding rate, acumulative number of data units send over a predefined frame range isdetermined. To start, a frame index is initialized to zero and a datacount is set to equal zero (step 1020). A video signal is received froma signal source (step 1022) and signal variations between successiveframes are determined (step 1024). Encoded data over a current frameperiod is determined (step 1028). The encoded data over the currentframe period is determined under a constraint of a permissible encodingrate received from the flow controller 260 (step 1026). The encoded dataover the current frame period is transmitted to a respective client 160(step 1030) and the frame index is increased by 1 (step 1032). The sizeof encoded data (in bytes for example) is added to the data count (step1034). If the frame index is less than the predefined frame range (step1036), step 1022 is revisited to continue receiving the video signalover a subsequent frame period. Otherwise, if the frame index equals thepredefined frame range, the mean bit rate over the frame range isdetermined (step 1038) as a ratio of data count (in bits) to duration ofa frame range. The mean bit rate is communicated to the overlay module230 (step 1040). The frame index is reset to equal zero (step 1042), tostart determining a data count over a new frame range, and step 1022 isrevisited to continue receiving the video signal over a subsequent frameperiod.

FIG. 11 illustrates allocation of notification-display intervals. Fivedisplay intervals 1112, 1114, 1116, 1118, and 1120 starting at timeinstants t₀, t₁, t₂, t₃, and t₀ ad having durations of δ₀, δ₁, δ₂, δ₃,and δ₄ are illustrated. A time instant t_(j), 0≦j<5, determines aposition within a frame, i.e., a position on a display screen, at whicha first point of a corresponding notification starts. A duration δ_(j),0≦j<5, determines an area within a displayed frame, i.e. an area on adisplay screen, occupied by a corresponding notification. The overlaymodule 230 superimposes notifications related to change of encodingrates during interval 1112, notification relevant to frame rates duringinterval 1114, notifications relevant to data loss during interval 1116,notifications relevant to transfer delay during interval 1118, andnotifications relevant to client's buffer occupancy during interval1120. The above intervals and corresponding notifications are exemplary.An arbitrary number of notifications and corresponding durations may beallocated subject to constraints regarding a permissible ratio ofnotification data to payload data. Each interval occupies a smallfraction, for example less than 0.01, of a frame period. Thus, for avideo signal, each notification occupies a small fraction of adisplay-screen area.

In one embodiment, the five intervals may be allocated in one frame andrepeated for several frames. For example, at a frame rate of 40 framesper second, allocating the display intervals in 80 successive framesresults in the notifications appearing on a display device (a screen)for two seconds. In another embodiment, the first display interval δ₀ isallocated in each of a first number of successive frames then the seconddisplay interval δ₁ is allocated in each of a second number of frames,and so on. For example, each of the five intervals may be individuallyallocated in 80 successive frames so that the correspondingnotifications would appear on a display device in five successiveperiods of two seconds each.

Several other variations of arranging the display intervals may bedevised. For example, two display intervals may be allocated in a numberof consecutive frames and three display intervals may be allocated in asubsequent number of frames.

The five notification types described above are only exemplary. It maybe desirable to limit the number of notifications related to connectionconditions to include, for example, notifications related only todata-loss and transfer delay. Thus, after allocating anotification-display interval to report encoding-rate change, anadditional notification-display interval may report both data-loss andtransfer-delay conditions.

Thus, a signal may be observed over distinctly defined successive timeintervals (for example time intervals of one second each) and feedbackinformation may be analyzed to extract a current data-loss fraction anda current transfer delay between the streaming server and the client.

A first relative difference between the current data-loss fraction and aprevious data-loss fraction and a second relative difference between thecurrent transfer delay and a previous transfer delay are determined. Ifan absolute value of either the first relative difference or the secondrelative difference exceeds a predetermined variation threshold (forexample of a relative magnitude of 0.2), then indications of the currentdata-loss fraction, the first relative difference, the current transferdelay and the second relative difference may be superimposed on thesignal during the additional notification-display interval. The value ofthe previous data-loss fraction is then updated to equal the currentdata-loss fraction and the value of the previous transfer delay isupdated to equal the value of the current transfer delay.

As defined above, an encoding coefficient, denoted E, is a scalingfactor which is multiplied by a nominal flow rate to determine apreferred flow rate compatible with a current state of a connectionbetween a streaming server and a client. A permissible value of anencoding coefficient E for a connection is determined according tofeedback information from a client. The encoding coefficient E isinitially set to equal 1.0, i.e., the encoding rate is the nominalencoding rate specified when the connection is initially setup. Theencoder 240 naturally produces data at rates which vary according to thenature of the encoded signal as illustrated in FIG. 8. The encodingcoefficient E, however, causes the encoder to independently modify itsmaximum output flow rate regardless of the dynamics (the fluctuating bitrate) of the encoded signal. The flow controller 260 extracts connectionmeasurements from the feedback data.

The mean values of measurements, over successive or overlapping timewindows, may be used to determine a new value of the encodingcoefficient E. A statistic based on a mean value of measurements takenover a time window and corresponding to a specific connectioncharacteristic, such as transfer delay or data-loss, is compared with apredefined acceptable reference value of the specific connectioncharacteristic. When the statistic exceeds the reference value by asignificant amount, the encoding coefficient is reduced and,consequently, the encoding rate is decreased below the nominal flowrate. If the statistic is below the reference value by a significantamount, the encoding coefficient may be increased. Thus, two bounds μ₁and μ₂, μ₁<μ₂, corresponding to the specific characteristic, may bedefined. The encoding coefficient is increased when the statistic isbelow μ₁ and decreased when the statistic exceeds μ₂. The selection ofthe gap (μ₂−μ₁) is critical. Avery small gap may result in unnecessaryrapid flapping between low values and higher values. A very large gapmay result in slow response, or even no response, to significantconnection-condition variations.

FIG. 12 illustrates data transfer from the streaming server 120 to aclient 160 using the Real-time Transport Protocol.

The streaming server 120 prepares the RTP packet (step 1220) and in step1222 a current value of the encoding coefficient is determined. In step1224 a sequence number of the RTP packet and a time of transmitting theRTP packet are recorded. In step 1226 the RTP packet is transmitted tothe client 160.

FIG. 13 illustrates a process of receiving RTCP (Real-time ControlTransport Protocol) packets from the client 160 and determining apermissible encoding rate accordingly. The process is activated in step1320. The flow controller 260 (FIG. 2) receives RTCP packets (step 1340)through network interface 250 and extracts measurement data from theRTCP packet (step 1342). A difference between a current time instant anda time instant of a preceding update of the encoding coefficient isdetermined in step 1344. If the time difference is less than apredefined minimum update interval, step 1340 is revisited to consider awaiting or forthcoming RTCP packet. Otherwise, if the time differenceequals or exceeds the minimum update interval, step 1346 determines anew permissible encoding rate. Step 1348 determines whether an update ofthe encoding coefficient is needed. If so, a new encoding coefficient isdetermined. Otherwise step 1340 is revisited. In steps 1360 and 1370,the encoding coefficient is updated and communicated to the encoder 240and to the overlay module 230 (step 1380). The flow controller 260 (FIG.2) is then ready to consider a new RTCP packet (step 1340).

FIG. 14 details step 1342 of FIG. 13. In step 1420, a received RTCPpacket is examined to determine if it contains a “receiver report” fromthe client 160. If the RTCP packet does not contain a receiver report,step 1440 is implemented to determine if the RTCP packet contains abuffer-occupancy report. If the RTCP packet contains a receiver report,step 1422 determines a transfer-delay as a time difference between thecurrent time of arrival of the RTCP packet and the time of transmittinga corresponding RTP packet. The corresponding RTP packet is the RTPpacket having a sequential number which matches a number indicated infield “extended highest sequence number received” of the RTCP receiverreport. In step 1426, a minimum transfer delay is determined as thelesser of the transfer delay calculated in step 1422 and a previousvalue of the minimum transfer delay for the connection. The minimumtransfer delay is initialized as an arbitrary large value. The minimumtransfer delay is retained for future use as a reference for gaugingfluctuating transfer delay. If step 1440 determines that the receiverreport does not contain a buffer-occupancy report, the process isdirected to step 1344 of FIG. 13. Otherwise, buffer-occupancy data areextracted in step 1442 and the process is directed to step 1340 of FIG.13

FIG. 15 provides an overview of steps of determining connection metricsand a corresponding encoding coefficient (step 1346 of FIG. 13). In step1520, the slope of a regression line of performance measurements takenover a time window is computed using any of known analytical methods.The gradient of the measurements near the end of the time window is alsodetermined. In step 1522 a statistic of data-loss over the time windowis determined. In step 1524, a preferred value of the encodingcoefficient, denoted E*, based on the regression-line slope and thegradient, is determined. In step 1526, a preferred value of the encodingcoefficient, denoted E**, is determined according to the data-lossstatistic determined in step 1522. In step 1528, a new encodingcoefficient E is selected as the lesser of E* and E**. Under favorableconnection conditions, the preferred encoding coefficient may be allowedto exceed 1.0, i.e. the encoder may produce a stream momentarily havinga flow rate (bit-rate) exceeding the nominal flow rate assigned to theconnection.

The encoding coefficient E just determined may be further modifiedaccording to occupancy of a buffer placed at the client-end of theconnection. Step 1532 directs the process to step 1348 (FIG. 13) ifbuffer-occupancy data is not available. Otherwise, step 1534 determinesif a statistic of buffer-occupancy measurements taken over the timewindow exceeds a predefined threshold. If so, and if the encodingcoefficient determined step 1528 exceeds 1.0, the preferred encodingcoefficient is reduced to 1.0 in step 1540 and the process is directedto step 1348 of FIG. 13. Otherwise, if the condition of step 1534 is notmet, step 1534 directs the process to step 1348 of FIG. 13. It is notedthat steps 1534 and 1540 follow a selected rule. Other rules governingthe use of buffer-occupancy statistics may be used.

Although specific embodiments of the invention have been described indetail, it should be understood that the described embodiments areintended to be illustrative and not restrictive. Various changes andmodifications of the embodiments shown in the drawings and described inthe specification may be made within the scope of the following claimswithout departing from the scope of the invention in its broader aspect.

What is claimed is:
 1. A streaming server comprising a master hardwareprocessor and a memory device storing processor executable instructionsorganized into instruction sets forming: an encoder devised to generatean encoded multimedia signal subject to a permissible flow rate; anetwork interface devised to transmit said encoded multimedia signal toa client device and receive feedback information from said clientdevice; a flow controller devised to: determine said permissible flowrate according to said feedback information; and generate notificationdata; and an overlay module devised to superimpose said notificationdata onto an input multimedia signal acquired from a signal source andpresented to said encoder.
 2. The streaming server of claim 1 whereinsaid input multimedia signal comprises a number of constituent signalsof different types and wherein said encoder comprises a set of encodingmodules, each encoding module devised to encode signals of a respectivesignal type.
 3. The streaming server of claim 2 wherein said overlaymodule comprises a set of overlay units, each overlay unit devised tosuperimpose a notification compatible with a specific signal type. 4.The streaming server of claim 3 wherein said each encoding modulecomprises: a tracking module devised to: receive a signal of a specificsignal type; determine a measure of signal variation over a designatedperiod of time; and report said measure to a respective overlay unit;and a stream-formation module devised to generate a data stream of saidsignal.
 5. The streaming server of claim 1 wherein said processorexecutable instructions further comprises a probing module devised to:generate probing packets for monitoring condition of a connection fromthe streaming server to said client device; and communicate said probingpackets to said flow controller and to said network interface.
 6. Thestreaming server of claim 5 wherein said flow controller is furtherdevised to categorize said feedback information according to differentperformance characteristics of said connection.
 7. The streaming serverof claim 5 wherein said flow controller is further devised to: determinequality metrics of said connection according to said feedbackinformation; and determine, for each constituent signal of said inputmultimedia signal, a permissible flow rate according to said qualitymetrics.
 8. The streaming server of claim 1 further comprising a sourceinterface devised to receive a multimedia signals each constitutingsignals of different signal types from a multimedia signal source. 9.The streaming server of claim 1 further comprising a selector devised toselectively direct each constituent signal of said multimedia signal toone of: said overlay module; and said encoder.
 10. The streaming serverof claim 1 wherein said master hardware processor comprises multipleprocessing units operating concurrently.
 11. A streaming servercomprising: a plurality of encoder modules, each encoder module devisedto encode a signal of a respective signal type, of a predefined set ofsignal types, to produce a respective encoded signal according to arespective permissible flow-rate; a plurality of overlay units, eachoverlay unit devised to superimpose a notification onto a signal of arespective signal type; a source interface devised to receive anincoming multimedia signal from a multimedia signal source, saidmultimedia signal constituting a set of constituent signals of saidpredefined signal types; a flow controller devised to determine apermissible flow rate of each constituent signal of said multimediasignal; a master hardware processor coupled to said flow controller andsaid plurality of encoder modules; and a network interface devised totransmit a composite multimedia signal combining encoded signals of saidset of signals and corresponding notifications to a client device. 12.The streaming server of claim 11 further comprising a probing moduledevised to: generate probing packets for transmission along a connectionfrom said streaming server to said client device; and communicate saidprobing packets to said flow controller and to said network interface.13. The streaming server of claim 12 wherein said network interface isfurther devised to: receive data-streaming requests from said clientdevice; receive response packets corresponding to said probing packets;and communicate said response packets to said flow controller.
 14. Thestreaming server of claim 13 wherein said flow controller is devised todetermine said permissible flow rate based on said response packets andcorresponding probing packets.
 15. The streaming server of claim 11further comprising a selector coupled to said source interface, saidselector devised to direct each constituent signal of said incomingmultimedia signal to one of: a respective encoding module specific to asignal type of said each constituent signal; and an overlay unitcommunicatively coupled to said respective encoding module.
 16. Thestreaming server of claim 11 further comprising a combining modulecoupled to said plurality of encoder modules, said combing moduledevised to interleave encoded signals of said plurality of encodingmodules to produce an outgoing multimedia stream presented to saidnetwork interface for transmission to said client device.
 17. A methodof data streaming, implemented at a streaming server comprising ahardware processor, the method comprising: receiving at a sourceinterface of said streaming server an incoming multimedia signalcomprising a set of constituent signals of predefined signal types;directing, at a selector coupled to said source interface, eachconstituent signal of said incoming multimedia signal to one of: arespective encoder module of a set of encoding modules; and an overlayunit coupled to said respective encoder module for superimposing arespective notification onto said each constituent signal; encoding, atsaid respective encoder module, said each constituent signal to producea respective encoded signal; transmitting, from a network interface ofsaid streaming server, combined encoded signals produced at said set ofencoder modules to a client device; and receiving feedback informationfrom said client device; and formulating said respective notificationaccording to said feedback information.
 18. The method of claim 17wherein said directing is based on a respective signal type of said eachconstituent signal, said respective encoder module being devised toencode signals of said respective signal type and said overlay unitbeing devised to superimpose a notification compatible with saidrespective signal type.
 19. The method of claim 17 further comprising:determining at a flow controller of said streaming server a respectivepermissible encoding rate for each constituent signal of said set ofconstituent signals based on said feedback information; and determiningsaid respective encoded signal according to said respective permissibleflow-rate.
 20. The method of claim 17 wherein said feedback informationcomprises measurements characterizing: a connection from said streamingserver to said client device; and condition of said client device. 21.The method of claim 17 further comprising storing, in a memory devicecoupled to said hardware processor, nominal values of operatingparameters comprising: encoding rates for each signal type of saidpredefined signal types; lengths of selected observation intervals foranalyzing said feedback information; duration of a notification for saideach signal type; and a specified period for periodic insertions ofnotification data for said each signal type.
 22. The method of claim 17further comprising: generating, at said set of encoder modules, specificnotification data related to temporal variations of flow rates ofconstituent signals of said set of constituent signals; andcommunicating said specific notification data to respective overlayunits coupled to said set of encoder modules.